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Introduction 


Program  evaluation  is  a term  which  has  been  bantered  around  a great  deal  in 
recent  years.  While  there  is  much  talk,  there  is  little  agreement  about  what  the 
concept  really  means;  there  is  agreement  that  it  should  be  done  but  little  agree- 
ment about  how.  Anderson,  et  al.  (1975)  identify  eight  characteristics  of 
program  evaluation  on  which  there  is  wide  although  not  universal  agreement: 
provides,  decision  making  information;  should  be  useful  for  program  improvement 
decisions;  should  be  completed  prior  to  making  decisions;  involves  a human  judg- 
mental process  following  program  evaluation;  should  consider  long-  and  short-term 
objectives;  should  consider  incidental  input  and  output  varialles;  the  model  or 
design  should  be  multivariate;  ard  data  collection  should  conform  to  accepted 
standards,  including  validity,  reliability  and  ethical  responsibility. 

The  current  emphasis  on  program  evaluation  grows  out  of  several  societal 
forces  including  accountability,  legislative  mandate,  and  genuine  interest  in 
program  improvement.  Providers  of  funds  are  demanding  that  agencies  and  organiza- 
tions , n both  the  public  and  the  private  sectors  of  the  service  delivery  system 


demonstrate  the  effectiveness  of  programs:  are  they  doing  a "good  job"  and 
spending  wisely  the  monies  they  receive  through  voluntary  contributions  or  through 
taxes.  As  money  becomes  more  scarce,  this  trend  can  be  expected  to  intensify. 
Mandated  accountability  is  now  included  in  most  federally  legislated  programs.  A 
notable  illustration  is  P.L.  94-142  which  requires  that  evaluation  procedures  be 
established  for  educational  programs  serving  handicapped  pupils  (DHEW-OE,  1977,' 
121a. 146).  Program  evaluation  is  viewed  as  an  appropriate  mechanism  for  determining 
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accountability.  In  addition  to  those  outside  forces  toward  accountability  demanded 
by  funders  and  legislators,  many  professionals  are  increasingly  concerned  about  the 
benefit  to  the  clientele  of  the  services  they  provide. 

The  purooses  of  this  paper  are  to  describe  one  design  used  to  evaluate 
educational  programs,  namely,  accreditation/certification,  and  to  describe  how  this 
design  is  applied  to  the  evaluation  of  educational  programs  for  visually  handicapped 
pupi Is. 


The  Accreditation/Certification  Design 

Two  major  preliminary  steps  precede  the  application  of  accreditation/certifi- 
cation design  to  any  educational  program:  the  development  of  standards  and  of 
instruments  to  assess  the  extent  to  which  the  institution  meets  these  standards. 

The  development  of  standards  involves  obtaining  consensus  from  the  field,  i.e., 
professionals  and  consumers,  about  elements  in  best  practice.  Standards  thus 
developed  may  not  be  ideal;  rather  they  set  minimum  acceptable  levels  of  perfor- 
mance. Furthermore,  obtaining  input  from  the  field  in  this  process  is  time 
consuming.  Major  benefits  do  accrue  to  those  involved,  however.  Professionals 
and  consumers  become  more  aware  of  current  forces  in  the  field  which  are  related 
to  standards  for  practice;  they  benefit  from  the  interchange  with  others  on  a 
common  area  of  interest;  and  they  have  an  opportunity  to  view  critically  prevailing 
practices . 

The  development  of  the  self-study  instruments  is  likewise  time  consuming  but 
usually  does  not  involve  the  numbers  of  persons  required  for  the  formulation  of 
standards.  Critical  decisions  about  the  instruments  include  the  most  appropriate 
format  to  be  used;  the  content  to  be  included;  and  the  most  effective  procedure 
for  assessing  the  extent  to  which  the  standards  are  met.  The  instruments  typically 
include  the  following: 
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yuidulines  for  preparing  a statement  of  philosophy  and  objectives; 

format  for  describing  the  population  being  served; 

identification  of  critical  elements  in  the  components  of  the  program; 

application  of  the  standards  to  the  program  components; 

evaluation  summary;  and 

plans  for  program  improvement. 

With  the  standards  and  the  instruments  for  a self-study,  an  institution  can 

evaluate  its  program  using  this  design.  Typically  this  involves  the  following 

steps  if  that  institution  is  seeking  accreditation/certification: 

1.  The  institution  signifies  an  intent  to  apply  for  accreditation  or 
certification  from  a specific  accrediting/certifying  body; 

?..  The  staff  of  the  institution  engages  in  a self-study  which  typically 
consumes  a minimum  of  six  months; 

3.  A team  of  professionals  and  consumers  makes  an  on-site  visit  to  the 
institution  to  validate  the  results  of  the  self-study  and  prepares 
a report  of  its  findings; 

4.  The  institution  prepares  a response  to  this  report; 

5.  The  self-study,  the  report  of  the  on-site  team,  and  the  institutional 
response  are  reviewed  ty  the  accrediting/certifying  agency  and 
assessed  on  the  extent  to  which  the  institution  meets  standards; 

6.  The  accrediting/certifying  agency  acts  on  the  institution's  applica- 
tion for  recognition. 

Advantages  and  Disadvantages 

The  accreditation/certification  design  is  primarily  summative,  that  is,  the 
evaluation  attempts  to  measure  the  overall  effectiveness  of  an  already  developed 
program  (Anderson,  et  al.,  1975).  To  a degree,  it  is  formative  in  that  is  provides 
information  for  program  improvement.  Webster  and  Stufflebeam  (1973)  list  the 
following  advantages  of  this  design: 

it  provides  feedback  for  program  improvement, 
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results  directly  influence  the  object  of  the  evaluation, 
the  methodology  is  well  established, 
the  scope  of  information  obtained  is  wide, 
reports  include  both  strengths  and  weaknesses, 

an  appeal  process  is  provided,  if  an  accrediting  agency  is  involved,  and 
it  is  a widely  implemented  design  for  educational  institutions. 

To  these,  we  add  that  benefits  accure  to  personnel  involved  in  the  process  as  a 

V. 

result  of  their  participation  in  the  self-study  (Scholl,  1971;  Howe  and  Fitzgerald, 
1977).  Further,  this  design  does  emphasize  the  qualitative  aspects  of  a program, 
but  this  may  also  be  considered  a disadvantage. 

Disadvantages  identified  by  Webster  and  Stufflebeam  (1978)  include: 
audiences  may  be  overly  confident  in  the  study's  validity, 
reliability  and  validity  of  information  are  usually  unknown, 
objectivity  is  usually  in  doubt, 

intrinsic  qualities  of  an  object  are  overemphasized, 

costs  to  the  institution,  boch  in  staff  time  and  financial  resources, 
are  high, 

safeguards  against  biased  or  unqualified  evaluators  are  weak,  and 
they  are  vulnerable  to  political  manipulation. 

Properly  implemented,  the  process  is  extremely  time  consuming.  Ideally  the  self- 
study  should  involve  the  entire  staff.  However,  the  benefits  of  a carefully 
executed  self-study  with  all  staff  participating  can  outweigh  this  disadvantage. 

In  addition,  the  design  usually  yields  little  quantitative  data  that  lend 
tnemsel ves  to  statistical  analysis.  Finally,  the  standards  and  instruments  designed 
for  the  self-study  may  not  reflect  "best  practice,"  they  become  out  of  date  because 
of  rapidly  changing  societal  forces,  and  they  are  not  easily  and  quickly  modified 
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and  updated.  The  process  of  standards  development  is  time  consuming  and  usually 
does  not  lend  itself  to  short-cut  procedures. 

The  general  acceptance  of  this  evaluation  design  in  education  probably 
indicates  that  the  advantages  outweigh  the  disadvantages. 

Application  of  the  Design  to 
Educational  Programs  for  Visually  Handicapped  Pupils 

The  application  of  the  accreditation/certification  design  to  program  evalua- 
tion of  schools  and  agencies  for  the  visually  handicapped  is  now  well  established. 
The  activities  of  the  National  Accreditation  Council  of  Agencies  Serving  the  Blind 
and  Visually  Handicapped  (NAC)  are  summarized  by  Roberts  (1979). 

In  the  1960's  when  standards  for  education  were  developed  by  the  Commission 
on  Standards  and  Accreditation  of  Services  for  the  Blind  (COMSTAC,  1966),  about 
half  of  the  school-age  visually  handicapped  children  attended  residential  schools 
(Jcnes  and  Collin'-,  1966).  The  trend  toward  initiating  programs  in  public  schools 
was  gaining  momentum  during  that  decade,  however,  and  currently  the  majority  of 
these  pupils  are  now  enrolled  in  day  programs.  Although  portions  of  the  standards 
and  self-study  instruments  available  for  residential  schools  are  appropriate  to 
day  programs,  there  is  a need  for  special  guidelines  and  self-study  instruments. 
The  project  described  by  Spungin  (1979)  is  one  development  designed  to  improve  the 
quality  of  programs  through  agreement  of  professionals  in  the  field  on  administra- 
tive guidelines  for  program  operation.  A related  effort  is  the  day  school  project 
which  is  described  below. 

In  1975  the  Standards  Committee  of  Association  for  Education  of  the  Visually 
Handicapped  selected  the  development  of  standards  for  day  programs  as  a priority 
area  and  committed  some  of  its  funds  for  this  purpose.  A survey  of  teachers  of 
the  visually  handicapped  was  conducted  to  identify  instructional  areas  that  should 
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be  available  to  visually  handicapped  pupils  but  are  not  included  in  the  typical 

school  curriculum.  Responses  from  42  teachers  employed  in  school  districts  in 

Wisconsin,  Maryland,  Texas  and  Florida  selected  these  areas  which  require  special 

emphasis  during  instructional  time  in  the  regular  classroom  or  special  instruction 

Physical  Education 

gross  and  fine  motor 

movement 

rhythm  and  games 

Vocational  Training 

Leisure  Time  and  Recreation 

Human  Sexuality 

Concept  Development 
spatial  relations 
categorization 
identification 

Communication 

Speech 

Handwriting 

Braille  (readiness  and  skills) 

Optacon 

Typing 

Indepdendent  Living 

Daily  Living  Skills 
Social  Skills 
Orientation  and  Mobility 
Safety  Awareness 

Special  Skills 
Abacus 

Visual  Efficiency  Training 
Sensory  Awareness  Training 
Map  Reading 

Reference  Materials  Skills 


The  National  Study  of  School  Evaluation  publishes  a series  of  self-study 
instruments  for  regular  school  programs:  Elementary  School  Evaluative  Criteria 
(iIjSc.,  19/J),  Jj^i^rJjjgh  School/Middle  School  Evaluative  Criteria  (NSSE,  1970), 
and  S_eco^y_^o^j^y^^tlve  Criteria:  Narrative  Edition  (NSSE,  1975).  With 
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fin.inci.il  support  from  the  National  Study  of  School  Evaluation,  we  conducted  a 
study  to  determine  (1)  whether  these  instruments  would  be  appropriate  for  day 
school  programs  as  basic  self-study  guides  and  (2)  whether  a supplement  for  the 
special  curricular  areas  would  be  necessary.  One  hundred  four  teachers  about 
equally  distributed  among  the  elementary,  junior  high/middle  and  secondary  level 
participated  in  this  study. 

In  general,  teachers  expressed  a high  level  of  satisfaction  with  the  three 
instruments.  No  curricular  area,  including  driver  training,  was  viewed  as 
inappropriate.  Needless  to  say,  teachers  recommended  only  the  theoretical  not 
the  practical  portions  for  driver  training. 

This  part  of  the  study  also  gathered  information  from  the  104  teachers  about 
the  impact  of  degree  of  vision  and  level  of  intellectual  functioning  for  the 
curricular  areas  identified  in  the  preliminary  survey.  Six  curricular  areas  were 
considered  as  vitally  important  fo*’  all  or  almost  all  visually  handicapped  pupils 
regardless  of  degree  of  vision  or  level  of  intellectual  functioning:  daily 
living  skills,  leisure  time  activities,  physical  education,  sensory  awareness 
training,  social  skills,  and  visual  efficiency  training.  Abacus,  braille,  Optacon, 
and  visual  efficiency  training  were  the  only  subject  areas  considered  as  exclu- 
sively the  responsibility  of  teachers  of  the  visually  handicapped;  all  others 
were  viewed  as  shared  responsibility  with  regular  classroom  teachers.  A more 
detailed  report  of  this  study  may  be  found  in  Scholl  (1973)  and  Scholl  and  Wo i hi 
(1978). 

On  the  basis  of  findings  from  this  study  a self-study  and  evaluation  guide 
for  day  programs  was  prepared.  The  open  narrative  format  of  the  National  Study 
of  School  Evaluation  instruments  (1970,  1973,  1975)  was  adopted.  The  guide  was 
designed  to  be  used  independently,  or  as  an  instrument  to  accompany  the  one(s) 
used  by  a regular  school  when  applying  for  accreditation  from  one  of  the  regional 


-8- 


accrcdi ting  agencies  or  as  an  instrument  used  by  a residential  school  conducting 
a day  school  component  when  applying  for  accreditation  from  the  National  Accred- 
itation Council. 

Curricular  areas  are  divided  into  three  groupings:  instructional  areas 
offered  to  all  pupils,  instructional  areas  typically  taught  by  special  personnel, 
and  nonacademic/extracurricul ar  services  and  activities. 

For  the  instructional  areas  offered  to  all  pupils,  evaluators  check  those 
which  are  taught  by  the  special  teacher,  those  taught  in  the  regular  class  and 
those  provided  in  both.  The  curricular  areas  include  sub-components  of  communica- 
tion skills,  science/mathematics,  social  sciences,  physical/health  education, 
fine  arts,  and  practical  arts. 

The  special  instructional  areas  which  are  typically  the  responsi bi 1 ity  of 
the  special  education  teacher  include  sub-components  of:  communication  skills 
(tactile,  auditory,  and  print  reading  and  writing),  special  aids  for  teaching 
(e.g.,  the  abacus),  concept  development,  orientation  and  mobility  training,  and 
independent  living. 

The  nonacademic/extracurricular  services  and  activities  are  those  mandated 
by  P.L.  94-142:  counseling,  health  services,  transportation,  informal  activities 
(recess,  lunch  room),  recreational  activities  and  clubs. 

Cone! usion 

Visually  handicapped  students  account  for  about  one  percent  of  all  the 
hardicapped  receiving  services  under  P.L.  94-142  and  P.L.  89-313  (HEW-BEH,  1977). 
Within  the  total  school  population,  they  account  for  slightly  less  than  .1 
percent.  It  is  easy  to  understand  why  they  are  frequently  overlooked  by  school 
administrators  pressured  to  service  the  more  vocal  and  numerically  larger  cate- 
gorical groups. 
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Professional  personnel  working  in  the  area  of  the  visually  handicapped  are 
likewise  relatively  few  in  number.  In  many  local  districts,  one  professionally 
trained  person,  who  may  not  be  qualified  in  the  area  of  the  visually  handicapped, 
is  expected  to  provide  services  to  two  or  three  low  incidence  groups  (the  physically 
handicapped,  hearing  impaired  and  visually  impaired),  all  with  unique  or  different 
educational  needs.  At  the  state  level,  administrative  and  supervisory  responsi- 
bility for  programs  for  the  visually  handicapped  is  often  assigned  to  personnel, 
not  qualified  in  this  area.  Inadequate  preparation  of  leadership  personnel  may 
easily  result  in  lowered  quality  of  services  to  pupils  in  local  schools. 

These  conditions  emphasize  the  need  to  develop  standards  for  the  service 
delivery  system  including  both  day  and  residential  programs  and  evaluation  instru- 
ments to  assess  the  quality  of  educational  programs  for  visually  impaired  pupils. 

The  application  of  standards  to  educational  programs,  both  day  and  residential 
through  self-study  instruments  is  one  way  to  insure  that  all  visually  handicapped 
pupils  will  achieve  the  full  educational  opportunity  goal  promised  them  under 
P.L.  94-142. 
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